1
Modifying the MARS Scale to Evaluate
Health Applications
Session #107, February 13, 2019
Speaker: Ryan Romero, MPH Student, University of Florida
Rick Kates, Clinical Assistant Professor, University of Florida
2
Ryan Romero, MPH Student
Has no real or apparent conflicts of interest to report.
Frederick Kates, PhD
Has no real or apparent conflicts of interest to report.
Conflict of Interest
3
1. Introduction to MARS
A. Sections of the MARS Scale
B. Application Specific Measures
C. Validation of Content Specific Measures
2. Study Specific Populations and Our Application of MARS
A. The deaf and Hard of Hearing Community
B. Use of Hard of Hearing Applications
C. App Specific Measures Developed for Target Population
D. Study Characteristics
3. New Criteria For Application Specific Measures Using MARS
A. Identification of a content expert in subject field (with example)
B. Use of content expert or population member to help develop content specific
measures
C. Use of content expert to gauge specific interests and needs of population
D. Short wrap up discussion
Agenda
4
Design content-specific criteria to evaluate mobile
health applications
Identify and engage an appropriate content
expert
Identify population-specific needs and implement
criteria into rating scale
Learning Objectives
5
Content Expert: Knowledge expert in a particular field of study
that has some professional involvement in the subject
[1]
Application Specific Measures (Content Specific): Evaluation
metrics designed to gauge the value of specific features towards
an intended population/ function
[2]
Appropriateness: Relevance of features to intended users
MARS (Mobile Application Rating Scale): Quantitative (1-5)
scale used to evaluate mHealth applications
[2]
Key Terms
6
Join at kahoot.it
1. Health Infinity 2. Carb Manager 3. MyNetDiary
Which mHealth App is the Best?
7
1. Health Infinity
8
2. Carb Manager
9
3. MyNetDiary
10
The Mobile Application Rating Scale (MARS)
[2]
Developed at Queensland University of
Technology
Stoyan Stoyanov, Leanne Hides, David Kavanagh,
Oksana Zelenko, Dian Tjondronegoro, Madhavan
Mani
Provides a standardized approach to comparing
and rating health applications for mobile devices
(iPhone, iPad, Android, Android Tablet)
Allows for modularity as key part of design
Frequently used to evaluate mHealth applications
Introduction to MARS
11
Sections of the MARS Scale
[2]
Engagement
5 subsections
Functionality
4 subsections
Aesthetics
3 subsections
Information
7 subsections
Subjective
Quality*
12
Alpha test of interrater agreement is preformed to
ensure acceptable levels of agreement
[3]
.
Sections where there is a lack of agreement or
uncertainty are discussed and resolved by
consultation of the content expert and
reevaluation.
Training and Validation of Raters
13
MARS allows for custom measures intrinsically
Tells us about app quality
Tells us about appropriateness and intended fit
Use of section as indicator of status vs indicator for change*
Section can be used to both address quality and declare the
need or desire for specific feature
Content development and developer feedback
Use of content expert to guide study’s intent
Feedback to developers
Empirical data to support stance
Content Specific Measures
14
Study Specific Populations
and Our Application of the
MARS
15
New Translation Technologies
16
The Deaf and Hard-of-Hearing community
- 7 million members of HOH community
[6]
Electronics as part of social interface capability
Electronics used to communicate with family and friends
Necessity for evaluation of hard-of-hearing applications
Hard of Hearing Apps & Our Study
17
Preliminary Literature Review
37 million with hearing difficulties
[7]
Needs of population over 60 years old
[8]
Decision to Include Content Expert
Inclusion and Exclusion Criteria
Need to determine appropriateness
[9]
Mitigate risk
Results
Study Characteristics
18
Application features
Is technology likely to be used?
Inclusion and Exclusion Criteria
Multi category approach to population specific applications
Text to Speech
ASL Translator
Hard-of-Hearing Assistant
Hard of Hearing Applications
19
Content Expert
Stephen J. Hardy, II
M.Ed.
Lecturer, University of Florida
College of Public Health and Health
Professions
Fluent in Deaf Culture, ASL, and Deaf
Studies
20
App-Specific Measure #1
21
App-Specific Measure #2
22
App-Specific Measure #3
[4]
23
App-Specific Measure #4
24
Related Study
[5]
25
5 / 21 applications no longer available for evaluation after follow
up
Simply disappeared
Indicator of status of application field
Potential lack of content validity
Niche product does not necessarily entail less competition
Application Turnover
26
Please take out your smartphone or device and enter the
activity at: Kahoot.it
Winner will receive small non-monetary prize to keep it
interesting
Time to Get Involved!
27
Three important topics:
1. Use of content experts as guide for project
2. Use of content experts to achieve validity of experiment design
3. Approaching a content expert for consultation
Content Experts and Design Validity
28
What qualifications should my content expert have?
How should they relate to the topic we are studying?
How can they be used to the advantage of the study?
Guide for purpose
Who does study effect?
Discussion section
Use of multiple content experts
Identification of a Content Expert
29
Knowledge on use of content experts in
experimental design and analysis
Recognition of importance of content
validity and designing a study to help the
subject it focuses on
Recommendation of quality apps based
on MARS
How Can This Information Help YOU?
30
Discussion on how this information relates to your focus
Using this information to develop a study
How health information professionals can use this
information to design better applications
Discussion and Q&A
31
Ryan Romero, MPH Student
ryan.romero1000@ufl.edu
Rick Kates, PhD
kates.rick@phhp.ufl.edu
Please feel free to come up and speak with us after the presentation!
Speaker Contact
32
1. Grant, J.S., & Davis, L.T. (1997). Selection and use of
content experts in instrument development. Research
in Nursing & Health, 20, 269274.
2. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M
Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps JMIR
Mhealth Uhealth 2015;3(1):e27. URL: https://mhealth.jmir.org/2015/1/e27
3. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull
19790101;86(2):420. [doi: 10.1037/0033-2909.86.2.420]
4. SkoogMusic. The Skoogmusic Skwitch. Digital Image. Accessed from
http://skoogmusic.com/skwitch/
5. Arellano, Patricia, Bochinski, Janet, Elias, Beth, Houser, Shannon, Martin, Thomas, Head, Hank.
2012. Selecting a Mobile App: Evaluating the Usability of Medical Applications. mHIMSS App
Usability Work Group.
6. Carroll DD, Courtney-Long EA, Stevens AC, Sloan ML, Lullo C, Visser SN, Fox MH, Armour BS,
Campbell VA, Brown DR, Dorn JM. Vital signs: disability and physical activity--United States,
2009-2012. MMWR Morb Mortal Wkly Rep 2014 May 9;63(18):407413. PMID:24807240
7. Ross, Mitchell, Karchmer, Michael. (2004). Chasing the Mythical Ten Percent: Parental Hearing Status of Deaf and Hard of Hearing Students in the
United States. Sign Language Studies, (4)2; 138-163.
8. Hoffman, H. J., Dobie, R. A., Losonczy, K. G., Themann, C. L., & Flamme, G. A. (2017). Declining Prevalence of Hearing Loss in US Adults Aged 20
to 69 Years. JAMA otolaryngology-- head & neck surgery, 143(3), 274-285.
9. Lewis, T. L., & Wyatt, J. C. (2014). mHealth and mobile medical Apps: a framework to assess risk and promote safer use. Journal of medical Internet research, 16(9), e210.
doi:10.2196/jmir.3133
References